Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eth: refactor tx sending #2979

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Conversation

buck54321
Copy link
Member

Refactor sending evm txs so that we store the txs before sending and are accepting of send errors. This avoids a scenario where we send the tx, but still receive an error, perhaps because of a bad connection or other problem with the rpc provider.

Implement a send queue so that the caller doesn't have to wait for the send. This should speed up ticks in core if the rpc provider is functional but slow to respond.

Refactor sending evm txs so that we store the txs before sending
and are accepting of send errors. This avoids a scenario where
we send the tx, but still receive an error, perhaps because of
a bad connection or other problem with the rpc provider.

Implement a send queue so that the caller doesn't have to wait
for the send. This should speed up ticks in core if the rpc
provider is functional but slow to respond.
Copy link
Member

@JoeGruffins JoeGruffins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Working well, but unsure if it solved the problem in core. I will still see the long delay warning with loadbot and some long blocking ticks.

A few errors while testing, but I dont know if because of the changes:

2024-09-18 11:01:47.309 [WRN] CORE: notify: |WARNING| (order) Match revoked - Match 47c635b0 has been revoked - Order: 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.309 [TRC] CORE: Not unlocking coins for order with status booked
2024-09-18 11:01:47.309 [WRN] CORE: Match 47c635b022252f880462a53c757a677dd1ef57131ab6189f06d4607aa4345d33 revoked in status MakerRedeemed for order 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.313 [WRN] CORE: Revoking order 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.314 [TRC] MUX: "POST http://127.0.0.3:5758/api/walletsettings HTTP/1.1" from 127.0.0.1:48026 - 200 40B in 71.033µs
2024-09-18 11:01:47.316 [ERR] CORE: notify: |ERROR| (order) Order revoked - Order 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35 on market eth_btc at 127.0.0.1:17273 has been revoked by the server - Order: 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.330 [TRC] MUX: "POST http://127.0.0.3:5758/api/walletsettings HTTP/1.1" from 127.0.0.1:48026 - 200 40B in 52.198µs
2024-09-18 11:01:47.632 [ERR] CORE: notify: |ERROR| (order) Redeem reporting error - Error notifying DEX of redemption for match 39bbc6c9399f2c418c126acaf98bf546910e12325b02bbabfb64fac81ff3ff83: error sending 'redeem' message: timed out waiting for "redeem" response (timeout) - Order: 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.632 [ERR] CORE: notify: |ERROR| (order) Redeem reporting error - Error notifying DEX of redemption for match 8f8a30a12a924a142a8557132218975a5cec36703711ea0a0b7f3e097726ad6e: error sending 'redeem' message: timed out waiting for "redeem" response (timeout) - Order: 4dd9852aefe0cf87b2d2f000a419f875fbe59275156fb2e6a3a5ba02c8bf4f35
2024-09-18 11:01:47.633 [ERR] CORE[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":87,"payload":{"result":null,"error":{"code":24,"message":"failed to find redeemed coin 0c8a532d34a2cd9e1f537b847d0b9ce0d99460cf12d3bd608b9629fe29d7dab4:0"}},"sig":""}
2024-09-18 11:01:47.633 [ERR] CORE[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":88,"payload":{"result":null,"error":{"code":24,"message":"failed to find redeemed coin 0c8a532d34a2cd9e1f537b847d0b9ce0d99460cf12d3bd608b9629fe29d7dab4:1"}},"sig":""}
2024-09-18 11:23:44.709 [TRC] CORE[btc][SPV]: PublishTransaction(87d1579da7113d02b6c965080a2ff8926059b2b7fdc1101dd978612f73641e09) completed in 50.992061935s
2024-09-18 11:23:44.932 [DBG] CORE: Waiting 3.7s to tick trade 68f6ea63d2145d3cb35ba611a7e6e951339ae9534a2520ba4ecb02335d9ed91c with 17 active matches
2024-09-18 11:23:44.932 [INF] CORE: runJob(redemption) completed in 1.249934129s
2024-09-18 11:23:44.932 [INF] CORE: Notified of maker's redemption (eth: 0x469a81253e1db943f47102496030820a13b3e70e4afdfe33ddd78601fb501231) and validated secret for order 68f6ea63d2145d3cb35ba611a7e6e951339ae9534a2520ba4ecb02335d9ed91c...
2024-09-18 11:23:44.937 [DBG] CORE: Unsubscribing from the eth_btc order book for 127.0.0.1:17273
2024-09-18 11:23:45.132 [ERR] CORE: Route 'epoch_order' notification handler error (DEX 127.0.0.1:17273): no order book found with market id "eth_btc"
2024-09-18 11:23:45.132 [ERR] CORE: Route 'epoch_order' notification handler error (DEX 127.0.0.1:17273): no order book found with market id "eth_btc"
2024-09-18 11:23:45.132 [ERR] CORE: Route 'epoch_order' notification handler error (DEX 127.0.0.1:17273): no order book found with market id "eth_btc"
2024-09-18 11:23:45.132 [ERR] CORE: Route 'epoch_order' notification handler error (DEX 127.0.0.1:17273): no order book found with market id "eth_btc"
2024-09-18 11:23:45.132 [INF] CORE: Notified of maker's redemption (eth: 0xa38a58763fbeb9e9f0b7cf07a83098e36e0a401d32d806a2c6a2629d2665336a) and validated secret for order 68f6ea63d2145d3cb35ba611a7e6e951339ae9534a2520ba4ecb02335d9ed91c...
2024-09-18 20:23:51.520 [TRC] MANTLE:STACKER:1: book note received
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for 4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258 succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for ed8c399a6bc8fba32ff3740d66a698ed4c4366d9a53577bef68a3eadb1859646 succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for 55ac8998a3554c56f9b09638dd94ed09fc67738abbacddcf3ae73603c8b0b17b succeeded
2024-09-18 20:23:51.521 [TRC] MANTLE:STACKER:1: book note received
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for f377f9cae7a8b7b49ba140663d4286ca510f0c954e10d4119dda6892c490835d succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for b9dc2ebf43dbcc91799af18edd46b93ccb4254b964a8903c4e6ea687bfdfa025 succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for 06d0d98d75b7f043eda075116fe83e3781ba084bd6200042e4478f8b34f6e3ed succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for 49949cc075b456d5f596c9a193cb036a490b0658d3dc03e4f9e7c9d0f827eb82 succeeded
2024-09-18 20:23:51.521 [DBG] CORE:STACKER:1: async processPreimageRequest for a86c81866a07465ab1e7e88a2e029d9a09eaebd738c6ac46dc416a070138870f succeeded
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33807,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33809,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33813,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33810,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [INF] CORE:STACKER:1: Received bondexpired notification from 127.0.0.1:17273 for account 40c3d950a8824878fcec1289d974e110359089dd08fbf640bff5fe37d22fa4c8...
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33820,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33815,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33808,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
2024-09-18 20:23:51.521 [ERR] CORE:STACKER:1[wss://127.0.0.1:17273/ws]: No handler found for response: {"type":2,"id":33818,"payload":{"result":null,"error":{"code":32,"message":"unknown response ID"}},"sig":""}
....
2024-09-18 20:23:51.522 [WRN] CORE:STACKER:1: Revoking order 4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258
...
2024-09-18 20:23:53.572 [ERR] CORE:STACKER:1: notify: |ERROR| (order) Order revoked - Order 4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258 on market eth_btc at 127.0.0.1:17273 has been revoked by the server - Order: 4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258
2024-09-18 20:23:53.572 [TRC] MANTLE:STACKER:0: book note received
2024-09-18 20:23:53.573 [CRT] MANTLE:STACKER:1: Error note received: {"type":"order","topic":"OrderRevoked","subject":"Order revoked","details":"Order 4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258 on market eth_btc at 127.0.0.1:17273 has been revoked by the server","severity":5,"stamp":1726658632756,"acked":false,"id":"295136668dd800f2","order":{"host":"127.0.0.1:17273","baseID":60,"baseSymbol":"eth","quoteID":0,"quoteSymbol":"btc","market":"eth_btc","type":1,"id":"4f697635e23ae63dc5bf09344d2e1a60e8a3a7b7bb4c8f6101437affa115f258","stamp":1726658596325,"submitTime":1726658592070,"sig":"3044022046a7a895373d257d64dff05f5c2e19da3f8b769a8d27e894d2ef72bba37d0a16022014206f1b20b76890f5d7ff205c6fc9105c34110061bc00b68abd1a5efc7ad2b9","status":5,"epoch":115110573,"qty":100000000,"sell":false,"filled":0,"matches":null,"cancelling":false,"canceled":false,"feesPaid":{"swap":0,"redemption":0,"funding":0,"refund":0},"allFeesConfirmed":true,"fundingCoins":[{"id":"d4474ff18b0575a97c9c78b138e86ee8865fee106a9a0576a157d9b9f88d0e6400000001","stringID":"640e8df8b9d957a176059a6a10ee5f86e86ee838b1789c7ca975058bf14f47d4:1","assetID":0,"symbol":"btc"}],"lockedamt":0,"parentAssetLockedAmt":0,"redeemLockedAmt":0,"refundLockedAmt":0,"accelerationCoins":[],"rate":658000,"tif":1,"targetOrderID":"","readyToTick":true}}

Comment on lines +1053 to +1055
select {
case qt := <-w.sendQ:
queued = append(queued, qt)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this only allows one qt to be queued at a time in the slice, so the length of queued will be 0 or 1 and you don't need a slice. Am I reading wrong? w.sendQ does still allow for a queue.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I did realize that I just hadn't gotten around to fixing it yet. I think I'm gonna refactor to get rid of the atomic uint32 too and just fall back to synchronous sending if the buffered channel is blocking.

Comment on lines +354 to +358
creds, err := walletCredentials(chainCfg.ChainID, walletDir, net)
if err != nil {
return nil, nil, nil, fmt.Errorf("error generating wallet credentials: %w", err)
}
cl, err := newMultiRPCClient(creds, providers, log, chainCfg, 3, net)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add logging

@buck54321 buck54321 marked this pull request as draft September 26, 2024 07:23
@buck54321
Copy link
Member Author

Moving to draft for now. I think there are some good things here, but it doesn't clearly help the issues we see with long ticks in core and there are some potential drawbacks that need more consideration.

One thing I'm very worried about is that we could successfully send a tx, but still see an error from the rpc server for whatever reason or just a poorly-timed disconnect. That would cause some issues up the line, and could be the cause of a couple of reported bugs where redemptions were not properly recorded in core.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants